The Impact of Image-Difference Features on Perceived Image Differences

We discuss a few selected hypotheses on how the visual system judges differences of color images. We then derive five image-difference features from these hypotheses and address their relation to the visual processing. Three models are proposed to combine these features for the prediction of perceived image differences. The parameters of the image-difference features are optimized on human image-difference assessments. For each model, we investigate the impact of individual features on the overall prediction performance. If chromatic features are combined with lightness-based features, the prediction accuracy on a test dataset is significantly higher than that of the SSIM index, which only operates on the achromatic component. Introduction A measure that accurately predicts perceived image differences would be useful for evaluating and optimizing image-processing methods, particularly in the areas of image compression, transmission, and reproduction. Various image-difference measures (IDMs) have been proposed in the literature. They are based upon hypotheses on distortions to which the human visual system (HVS) is particularly sensitive. Such hypotheses include pixelwise color differences, pixelwise differences of color attributes (i.e., lightness, chroma, and hue), differences in spatial changes of color attributes (gradients or variance), color differences within large areas of the same color [1], global and local contrast changes, changes of structural information [2], and differences in special regions of interest (edges, corners) [3]. Although the cortical mechanisms of image-difference perception are poorly understood, existing IDMs quite successfully predict perceived image differences for many individual distortions such as compression artifacts, noise, and blur. Most methods also incorporate hypotheses on how the HVS judges combinations of distortions that occur, for instance, in gamut mapping or tone mapping. Nevertheless, the prediction performance of IDMs for images with such complex distortions can be improved. In this paper we analyze the influence of several hypotheses on the perceived overall image difference. Assumptions about how the HVS combines individual differences into an overall image difference are also investigated. We utilize the recently presented image-difference framework [4] for our evaluations. In addition, we describe in detail how our IDMs are derived from the hypotheses. The Image-Difference-Feature Framework The visual mechanisms that contribute to the assessment of image differences are very complex. We attempt to model these mechanisms empirically as a combination of simple hypotheses on how the visual system performs such tasks. These hypotheses are mathematically expressed as image-difference features (IDFs). Given two input images, our framework computes several IDFs, which are then combined to optimize the prediction of human image-difference assessments [4]. As a first step of our IDF computation we use an imageappearance model to normalize the images to specific viewing conditions (illuminant, luminance level, viewing distance). They are then transformed into a working color space with certain beneficial properties — for instance, Euclidean distances in this space should closely match perceived color differences. This ensures that image features, such as edges and gradients, are judged correctly by the subsequent feature-extraction routines. In summary, image-difference features are computed as follows: 1. Image-appearance-model transformation The input images are transformed by an image-appearance model to normalize them to specific viewing conditions. Here, we use S-CIELAB, which filters the images “to simulate the spatial blurring by the human visual system” [5]. 2. Color-space transformation The images are transformed into a working color space. In this paper, we use the LAB2000HL color space [6], which is both hue linear and highly perceptually uniform with respect to the CIEDE2000 color-difference formula [7]. 3. Difference-map generation Maps are generated that reflect differences between the images, e.g., gradient differences or color differences. 4. Characteristic-value computation Each map is finally transformed into a characteristic value, e.g., the mean value of all pixels. The resulting IDFs are combined into an image-difference measure (IDM) using, e.g., an additive combination model. The coefficients of the model can be optimized on an image database with image-difference assessments of human observers. Distortions Affecting Image Differences Hypotheses of Distortions In the following, we present a list of low-level hypotheses investigated in this paper. We also explore their importance for the overall image-difference impression. More hypotheses, especially on high-level (semantic) image differences, can be found in the literature. Hypothesis 1: The HVS is sensitive to lightness, chroma, and hue differences. This, of course, includes color differences that are combinations of color-attribute differences. Gamut-mapping algorithms, for instance, incorporate this assumption [8]. CGIV 2012 Final Program and Proceedings 43 Hypothesis 2: The HVS is sensitive to achromatic contrast differences. Hypothesis 3: The HVS is sensitive to achromatic structural differences. This is particularly important if the distorted image contains artifacts such as banding. Hypothesis 4: The HVS is more sensitive to contrast changes in low contrast image regions. This hypothesis is related to Weber’s law, stating that threshold difference is proportional to the magnitude of the stimulus (in our case contrast). This phenomenon can also be observed for suprathreshold contrast differences: if, for instance, the contrast of a distinct edge is noticeably reduced, the edge information is still present; the same contrast change applied to a weaker edge may cause it to vanish. The latter contrast change is likely to be considered more disturbing. Hypothesis 5: The HVS is not equally sensitive to differences described in hypothesis 1–4 on different spatial-frequency bands. For reasons of simplicity we do not consider this hypothesis here. It can be incorporated into the analysis by a multi-scale approach. Derived Image-Difference Features A widely used image-difference measure is the structural similarity (SSIM) index proposed by Wang et al. [2]. It incorporates hypotheses 1 (at least for luminance differences), 2, and 3 by applying three functions in a sliding window centered around corresponding pixels x and y of two images: luminance: l(x,y,c1) = 2μxμy+ c1 μ2 x +μ 2 y + c1 , (1) contrast: c(x,y,c2) = 2σxσy+ c2 σ2 x +σ 2 y + c2 , (2) structure: s(x,y,c3) = σxy+ c3 σxσy+ c3 , (3) where μx and μy denote the means, σx and σy the standard deviations, and σxy the mean-adjusted inner product of the pixel values within the window. The variables ci > 0 serve to avoid instabilities for small denominators. The variable c2 of c(x,y,c2) can be adjusted to account for the contrast-masking property of the HVS described in hypothesis 4. These functions are combined using a factorial model, resulting in SSIM(x,y) = l(x,y,c1) 1c(x,y,c2) 2s(x,y,c3) α3 , (4) where c1, c2, and c3 are adjusted to the working color space, and α1, α2, and α3 may be used to weight the contribution of each function to the overall image-difference prediction. The SSIM index is a simple IDM and incorporates many of the hypotheses described above. In addition, a multi-scale extension was proposed, which incorporates hypothesis 5 [9]. Due to its advantageous properties and its popularity we create image-difference features based on the terms of the SSIM index. Comments on Image-Difference Features Before we start our investigations, we address some issues of using the SSIM functions in our framework: 1. The SSIM index is designed for grayscale images and ignores color information. To fully incorporate hypothesis 1, ad0 100 0 100 1